Composite Fermions and Quantum Hall Stripes on the Topological Insulator Surface 
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We study the problem of a single Dirac fermion in a quantizing orbital magnetic field, when 
the chemical potential is at the Dirac point. This can be realized on the surface of a topological 
insulator, such as Bi2Se3, tuned to neutrality. We study the effect of both long range Coulomb 
interactions (strength a = g) P ) and local repulsion U which capture the effect of electron cor- 
relations. Interactions resolve the degeneracy of free fermions in the zeroth Landau level at half 
filling, but in a manner different from that in graphene. For weak interactions, U — and a <C 1, a 
composite Fermi liquid is expected. However, in the limit of strong local correlations (large U but 
a <C 1), a charge density wave phase is predicted, which we term "axion stripe". While reminiscent 
of quantum Hall stripe phases, its wavelength is parametrically larger than the magnetic length, 
and the induced fermion mass term (axion) also oscillates with the charge density. This phase is 
destroyed by sufficiently strong Zeeman coupling. A phase diagram is constructed and consequences 
for experiments are discussed. 
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There has been much recent interest in topological in- 
sulators (for reviews see [HQ) with strong spin-orbit in- 
teractions, which feature protected surface states in the 
presence of time-reversal invariance (TRI). A minimal 
model of the surface states of a three dimensional strong 
topological insulator(TI) is a single Dirac node, which is 
impossible to obtain in a 2D band structure with TRI. 
Photoemission experiments on e.g. Bi2Se3 reveal pre- 
cisely such a surface band. These surface states may be 
considered as l/4th of graphene, the other well known 
Dirac metal, which has additionally two fold valley and 
spin degeneracies. 

A key feature of Dirac metals is their anomalous re- 
sponse to a magnetic field, which includes large Landau 
level (LL) spacing and an unusual offset in quantum Hall 
states. For a single Dirac node at neutrality (chemical 
potential at the Dirac point), the zeroth LL is half filled, 
implying an enormous degeneracy in the clean limit. In- 
teractions are critical to lifting this degeneracy. The pre- 
cise nature of the resulting state is the subject of this pa- 
per. When graphene is tuned to neutrality in a magnetic 
field, this Landau degeneracy is resolved by spontaneous 
symmetry breaking - two of the four LLs are chosen to 
be completely filled and a translationally invariant state 
is obtained. In the case here with a single Dirac node, 
this is not an option - the only ways for the system to lift 
the degeneracy is by forming a homogeneous fractional 
quantum hall (FQH) phase, or breaking the translational 
symmetry to form spatial patterns as in the stripe phase. 
Recently, Dirac like LLs on the surface of Bi2Se3 were ob- 
served 0, 3 ■ Although the chemical potential there was 
not tuned to neutrality, and the LL widths indicate the 
importance of disorder, this is a hopeful sign for future 
experimental work along the direction of this paper. 

The short range interactions of the TI surface modes 
will be parameterized by U, and the long range 



Coulomb interactions by the dimcnsionless coupling a = 
e 2 /(ehvF)- Here vp is the fermi velocity and e = 1+t ^ ak 
the effective surface dielectric constant. We will assume 
a <C 1 throughout, which allows us to treat the Coulomb 
interactions last. This is reasonable in currently known 
TIs since f > 1 (e.g., tbulk = 290 @] in Bi 2 Te 3 ) while 
D F ~3x 10 5 m/sec in Bi 2 Te 3 @, 0], thus a ~ 1CT 1 to 
10~ 2 . Estimating U is hard, but it is expected to be 
large in more correlated systems like the Iridates [H Q . 

In zero magnetic field, when the surface is at neutral- 
ity (recent experiments reported control over the surface 
chemical potential (loj ) , a metal to insulator transition is 
expected when U is varied across a critical value U c , due 
to the spontaneous generation of a magnetic mass term 
to. In an orbital magnetic field, our results are as follows. 
When a = 0, we find within a Hartree-Fock (large- N) ap- 
proximation that phase separation occurs for the entire 
range of U, into a pair of domains with completely filled 
and empty zeroth Landau level. On introducing weak 
Coulomb interactions a <C 1, macroscopic phase separa- 
tion is forbidden [ll|. In the following two regimes the 
resulting ground state may be predicted reliably. For 
U <C U c , a homogenous phase, the Halperin-Lee-Read 
composite Fermi liquid(CFL) is expected. In the oppo- 
site limit, for U > U c , a charge density wave (stripe) is 
obtained where the mass term "to" is also modulated. 
We term it as "axionic stripe" phase because in contrast 



to usual quantum Hall stripes [12 -[Hi, the wavelength A 
here is parametrically different from Is, the magnetic 
length. Indeed, A ~ lg(y/a£ i )~ 1 , where £ is the cor- 
relation length in the insulating phase. Other possible 
intermediate phases are also discussed. The results are 
summarized in Figure [1] The Zeeman coupling, when 
small, does not modify the phase diagram. However, if 
it is sufficiently large, it can destroy the "axion stripe" 
phase. The critical Zeeman coupling (or g factor: g = g c ) 
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FIG. 1: Zero-temperature phase diagram of a single Dirac 
node, in an external magnetic field, with a tunable short-range 
repulsion U and (a) no long-range Coulomb interaction, or (b) 
with a fixed weak long-range Coulomb interaction a<l. In 
both (a) and (b), we assume the chemical potential is at the 
Dirac node. Region I: quantum critical regime with Ib <C £. 
Region II: weak field limit in the magnetic ordered phase, 
namely magnetic length Ib is much greater than any intrinsic 
length scale. Region III: weak field limit when !b > ^ and 
U < U c - Region IV: Weak field limit when Ib 3> £ and 
U < U c , and also extended over the whole weak U regime. 
Phases predicted in regime I and III are valid in large- TV limit. 
Phases predicted in regime II and IV are valid for arbitrary TV, 
including TV = 1 (allowing only for flavor symmetric states), 
(c) Schematic picture of phase separation realized in region 
(a). 



depends on microscopic details such as the ultraviolet 
cutoff, but typically g c ~ 0(1). Finally, we point out 
that coating the available TI surfaces with a magnetic 
layer could also lead to a stripe state. 

Model with short-range repulsion: The effective theory 
of the surface states in an external magnetic field is: 



(1) 



where T-L is hamiltonian density, li = p + A, U = Ua 2 is 
the short-range repulsion when a lattice constant set to 
one for simplicity (this is the only possible four-fcrmion 
term without derivative for a single Dirac fermion), and 
& = [pxi^y] spin Pauli matrices. In this paper, we set 
vp, h, e, and c to one unless stated otherwise. Note that 
Zeeman coupling is ignored above but will be included 
later. For B = with weak interactions a Dirac semi- 
metal is obtained, since U is irrelevant by power counting. 
A metal-insulator transition is expected when U > U c . A 
unique feature of the current system is that the insulat- 
ing phase must break time-reversal symmetry, namely a 
magnetic mass term imp* a z tp is generated. Such a mass 
generation mechanism, if continuous, is described by the 
Gross- Neveu universality [l5[. We first study this phase 



transition and the magnetic field effect at the Hatree- 
Fock (or mean-field) level, which is essentially equivalent 
to a large-TV approximation(see suppl. mat.) By the 
mean-field (MF) approximation, 



H M f = ^ [n • 3 + ma z ]ip - 
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where m and p are the variational parameters; p = 
(tp^tfj — 1) and to = —U{ifra z i}) are the self-consistent 
conditions to minimize energy. Note that p is the elec- 
tron density with respect to half-filling. 

For a homogeneous m, the energies of LLs are: 



sgn(n)- v /2|ra|.B + m 2 , 



if n ^ 
if n = 0, 
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where B > (along positive z direction) is assumed. 
LL degeneracy is Close to charge neutrality all the 
negative LLs are fully filled and the zeroth LL is partially 
filled, p = (—■¥-) for the fully filled (empty) zeroth 
LL. The MF ground state energy is: 
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where £ is the generalized Riemann-Zeta function. The 
derivation is in the supplemental material. Similar re- 
sults have been obtained in Ref. [16[. We point out that 



the coefficient of the second term in our result is twice of 
the previous incorrect result, which is essential to obtain 
the right phase diagram. Here we explain the physical 
consequences. For B = and p = 0, the quadratic term 
is 8m 2 with S = 1/(2(7) — A/ (47r), from which we obtain 
the critical value U c = 2tt/A [corresponding on-site re- 
pulsion on a lattice is 27r/(Aa 2 )], at zero field. The mass 
correlation length £ is £ ~ 1/8. For finite B, the key 
feature of Eq. |4]is the presence of the linear term —mp, 
whose form is dictated by the symmetry argument below. 
Consider the time-reversal (TR) symmetry and the 

PH 

(unitary) particle-hole (PH) symmetry defined by ip a — > 
[cTxlapipp. The massless Dirac theory as well as the in- 
teractions respects both of them. Even though exter- 
nal magnetic field B — > —B under each of these, it rc- 
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spects the combination TR o PH, namely B 

TRoPH , TRoPH 

-p, we conclude that 
which dictates that the 



Since to — > —m and p 
E(p,m;B) = E(-p,-m;B), 
only possible linear term is —mp. 

The linear term —mp has important consequences. At 
half-filling, p = everywhere if the system is homoge- 
neous; consequently the system cannot take advantage 
of the linear term to gain energy. To have a finite p to 
gain energy from the linear term, the system can break 
translational symmetry either microscopically by devel- 
oping a density wave or macroscopically by phase separa- 
tion. The latter is realized with short-range interactions, 
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but we will see below that on including the long range 
Coulomb repulsion, a density wave can result. Note, 
since TR o PH is a symmetry of the Hamiltonian [Eq. 
(UJ) ] at half-filling, phase separation or a charge density 
wave(CDW) which involves generating a finite m or p 
spontaneously breaks the TRoPH symmetry. 

We denote the optimal mass as m* if the system is 
homogeneous i.e. p = everywhere. From Eq. [H m* = 
for U < U C (B) = [U- 1 - iCCDlWVW]" 1 . To 
sec whether the system tends to phase separate into two 
regions with ±(m, p), we examine the variational energy 
Eq. ((4]) around m* = by setting p = m/U <C 1: 



E(p,m;B) - E(0,0;B) = - 



m 1/2 c A, a 



3 ,(5) 



where higher order terms in m has been neglected. We 
find that phase separation occurs when a < Ib, which 
is always satisfied for currently available magnetic fields. 
For U > U C (B), m* ^ 0. In this case, the system always 
has phase separation tendency because the energy gain 
by forming ±(m*,p) is — \m*p\ which dominates energy 
cost Up 2 /2 for finite but small enough p. What is the 
optimal value of p to minimize the energy? It is tedious 
but straightforward to show that the optimal p = ±j^-, 
namely the zeroth LL is either fully filled or empty. Thus 
the two domains are quantum Hall states with v = 
with a chiral edge mode between them. 

Below we demonstrate that in some regimes phase sep- 
aration is reliable even beyond the MF level. First, in 
region II of Fig. [Ha), we are m massive phase with a 
weak magnetic field. Let us denote the optimal mass 
as = ±|m*|. Due to the linear term — pm*, the 
phase separation energy gain is — which domi- 

nates any other regular B 2 energy cost for small enough 
B. Second, in regime III, U < U c and magnetic field 
is weak. The Rcnormalization Group (RG) flows to the 
free Dirac fermion fixed point (FP) where perturbativc 
treatment is reliable. This dictates that the energy gain 
of phase separation in region III scales as ~ —^B 2 : where 
£ is the correlation length at zero magnetic field that di- 
verges at criticality. This again dominates any regular 
B 2 cost when U close to U c . Finally in the quantum 
critical regime (I), h < £, m ~ B 1 / 2 and the energy 
gain ~ — B z l 2 which also dominates sub-leading regular 
B 2 term. The arguments of phase separation for regime 
II only rely the gaussian massive FP, which is valid for 
arbitrary N including N = 1. The sign of the phase 
separation energy gain in regimes I, III, however, is only 
reliable in the large- iV limit, (see suppl. mat.) 

Coulomb Interactions: What happens if a weak long- 
range Coulomb interaction is included? First, macro- 
scopic phase separation is no longer possible 



111 be- 



cause of the infinite charging energy cost. Thus, either 
a homogeneous system or alternating spatial patterns of 
v = ±1/2 regions with characteristic pattern size l p will 
result. If eventually l p 3> Ib,£,, our previous calculations 
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FIG. 2: (color online) Variational results of the most likely 
patterns: Stripe (blue) and triagular lattice CDW (magenta). 
In the triangular lattice CDW phase we use the simple ansatz 
where the negative mass domains (black regions) form circles. 
Close to half-filling the optimal pattern is stripe. 



can still be trusted within each region, which allows us to 
treat Coulomb interaction as a perturbation classically: 
Hi 



Coulomb 



/ dr 2 dr 



a (P{r))(p(r')) 



We show that this 
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is indeed the case in regime I and II. 

By dimensional analysis the Coulomb energy density 
cost is ~ 4?r- The energy density cost of the domain 

wall separating different regions is ~//l p where 7 is the 
domain wall tension. By minimizing the total energy 
cost, the optimal l p is found to be l p ~ y/ejls with the 
optimal energy density cost ~ \f^jl 2 B . Note that this 
dimension analysis is independent of the exact shape of 
patterns, which will be considered shortly. 

In quantum critical regime (I), 7 ~ 1/7^ by hy- 
perscaling hypothesis. Thus, l p ~ y/els Ib when 
E=l/a>l. The total energy density cost is ~ y/a/lg, 
much smaller than phase separation gain ~ 1/^- In 
regime II, l B > £. When U is close to U c , 7 - l/£ 2 by 
hyperscaling hypothesis and the resulting optimal pat- 



tern size L 
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^> Ib- The optimal energy den- 



sity cost ~ y/a/{l 2 B C) is again much less than the phase 
separation gain ~ D ee P in the ordered phase 

U » U c , y/j and m are high energy scales where short- 
range physics is important. Their ratio is a non-universal 
number which generically is of order one. Comparing en- 
ergy cost ~ y/aj/l'g and gain ~ mjl 2 B it is clear that the 
ground state is also in stripe phase when a <C 1. 

What is the spatial pattern? The most natural options 
are stripes or a triangular lattice similar to a Wigncr 
crystal. As this is completely determined by the classical 
Coulomb interaction, we perform a variational calcula- 
tion with H Coulomb an d domain wall tension 7. We work 
in thin domain wall limit (domain wall width <C l p ) and 
carry out numerical Ewald-Summation to determine the 
precise coefficient in the Coulomb energy density ~ 
for each trial ansatz. The coefficient in the domain wall 
energy density ~ ^- is determined by geometry straight- 
forwardly. The optimal l p and energy cost is found by 
minimizing the sum of the two. As a function of aver- 
age v, we find the phase diagram in Figf2] When v w 
the optimal pattern is stripe; when the charge unbalance 
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(the area unbalance between v — 1/2 and v = —1/2 re- 
gions) is increased, a first order transition is found at 
v m ±0.15 into the triangular lattice charge density wave 
(CDW). (A narrow region of intermediate phases is ex- 
pected around this putative first order transition (l7}.) 
This filling dependence of the CDW pattern is similar to 
the one in the high LLs of a parabolic band (l2- 14|. 

Finally we discuss regime IV in FigQJb). In this weak 
field limit the short range U is irrelevant but Coulomb 
interaction is marginal. Thus the Coulomb interaction 
always dominates and we can ignore U . If LL mixing is 
ignored when a«l, because the wave functions in the 
zcroth LL of a Dirac metal arc the same as the lowest 
LL of a quadratic band (see suppl. mat.), the same set 
of Haldane pseudo-potentials is obtained for both cases, 
indicating the ground state in a CFL phase as confirmed 
by exact diagonalization numerics jl8j | . 

What happens if LL mixing is included? This is a 
rather neat problem because LL mixing is independent 
of magnetic field and only controlled by dimensionlcss 
constant a. We perform a standard second-order pertur- 
bation study by integrating out all the nonzero LLs in a 
single step, which gives corrections to the interaction in 
the zeroth LL. (See suppl. mat. Similar study has been 
performed on the LL mixing of quadratic band [l~9| .) We 
find the correction to pseudo-potential in the 0th LL is: 

Vi = 0.443 - 0.0068a, V 3 = 0.277 + 0.0028a. (6) 

This is a very small effect. It is worth noting that the 
direction of the first order correction is towards a Pfaffian 
phase and a stripe phase eventually fljj . 

Zeeman Splitting: We comment on the effect of Zee- 
man coupling which we have ignored so far. For the 0th 
LL, its electron spin is fully polarized along — z direction 
such that its Zeeman energy is a constant no matter how 
its electrons are distributed spatially. The Zeeman cou- 
pling is only through higher LLs. As shown in the suppl. 
mat., with only short-range U, phase separation is still 
favored in the MF approximation as long as U is not 
too strong compared with U c . On introducing the long 
range Coulomb interaction however, the "axionic stripe" 
phase in regime II may be destroyed by the Zeeman cou- 
pling. A simple estimate of Zeeman energy density is 
gfj,BB(ip' cr z ip) = guBBjj. Comparing with phase sepa- 
ration energy gain one finds that the condition for 
the "axionic stripe" phase to survive is 47r ^ MB < 1 (we 
put back units. U > U c and U c ~ hvp/A). This ratio is 
material dependent and can be of order unity (see suppl. 
mat.). Thus, we conclude that for materials with small 
g and/or large vp,a, Zeeman energy is only a small and 
negligible perturbation in the full phase diagrams of Fig. 
CD On the other hand if > 1, the "axionic stripe" 

phase in regime II of FigQJb) will be destroyed but other 
regimes are still valid. Which phases would then take its 
place is presently unclear, and left to future work. 



The short range U is likely to be smaller than U c in cur- 
rently known TIs, which are then likely in a CFL phase 
when the fermi level is close to the Dirac node and ex- 
ternal magnetic field is applied. How to realize the pre- 
dicted stripe phase? One way is to search for TIs with 
relatively strong U such that U > U c ; another way is 
to coat the surface with ferromagnetic (FM) materials 
such as Mn or Fe while keeping the surface states close 
to half-filling. For the latter, a magnetic mass is induced 
on the surface states of TI via exchange. Then the sys- 
tem can be effectively treated as in regime II and the 
stripe phase is preferred as a ground state when Zeeman 
coupling is small. The stripes and associated chiral edge 
modes along domain walls can be directly detected by 
microscopic probes such as STM. 

At what temperature can those predicted phases in 
FigH] be realized? For the CFL phase in regime IV, 
its signature of can be detected only if the tempera- 

2 

ture ksT <C -ffe, which is quite low given the large e 
of the current systems. For macroscopic phase separa- 
tion, they have a lower entropy density compared with 
the half-filled LL and the difference can be estimated to 
be fcs^:ln2 ~ by counting the number of choices 
to half fill the zero LL. Thus the phase separation crit- 
ical temperature T c ~ e &^b wncrc E ga [ n is the phase 
separation energy density gain. In the quantum critical 
regime I: T c ~ hv f which is of same order of the LL 

° c k B iB 

spacing. In regime II: T c ~ |^ when U > U c and T c 
becomes a high-energy scale ~ A in the deeply ordered 
phase. We conclude that phase separations in regimes I 
and II are very stable to finite temperature. Finally for 
the weak field phase separation regime III: T c ~ , 
With Coulomb interactions, if underlying lattice is ig- 
nored, the stripe phases breaks continuous translational 
symmetry which would be restored in 2D at any finite 
temperature. However, because of the lattice as well as 
disorder, we expect that stripes is pinned and become 
static which is amenable to STM investigations (l7j . 

Conclusion: We have argued for the existence of a com- 
posite fermi liquid at small U and a stripe phase for large 
U , in the presence of Coulomb interactions and a quantiz- 
ing orbital magnetic field. The phase structure for inter- 
mediate U is difficult to predict reliably, although large- 
N calculations predict a stripe phase there as well. This 
regime deserves further study, since it could harbor in- 
teresting homogeneous phases, like the Read-Moore state 
which is natural at this filling, with large characteristic 
energy scales. We hope these results will encourage ex- 
periments and the search for topological insulators with 
strong correlations. 

We thank Dung-Hai Lee, Steve Kivelson, and Jason 
Alicca for helpful discussions. YR is supported by the 
start-up fund at Boston College. HY and AV are sup- 
ported by DOE grant DE-AC02-05CH11231. 
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Supplemental Material 



Charge neutral point of an inversion symmetric 
perfect topological insulator 



In the majority of our study we assume the chemical 
potential is at or close to the Dirac node of the surface 
state. This is certainly possible if the sample is gated or 
doped so that the chemical potential is tunable. Here we 
try to further point out that for a perfect sample of a 
topological insulator with inversion symmetry, which is 
actually respected by all the current available experimen- 
tal systems of topological insulators such as Bii-^Sbz, 
Bi2Sc3 and Bi2Tc3 have inversion symmetry, our assump- 
tions is automatically realized. 

Consider a slab of topological insulator with open 
boundary condition along the z-direction. One can view 
a whole chain of unit cells along the z-direction as a sin- 
gle enlarged unit cell. In terms of these enlarged unit 
cells the system becomes two dimensional whose fermi 
surface states are those surface states. By inversion sym- 
metry the top surface and bottom surface fermi surface 
volumes must be identical. And by Luttinger's theorem 
band insulators should have total fermi surface volume 
zero. This dictates only two possibilities: the surface 
fermi volume is zero or one-half of the Brillouin Zone. 
The second case is more subtle and related to an addi- 
tional topological index of the polyacetylene-type: The 
edge states of polyacetylene chains along z-direction has 
electron charge ±1/2. Therefore the additional band 
forms a fermi surface volume of half the surface Brillouin 
Zone at half-filling. In the present study we assume the 
TI has trivial polyacetylene-type index and chemical po- 
tential is at Dirac node. 



in an external magnetic field is given by 



Hq + H int . 



(U = p+-A) (7) 



We abbreviated spin indices. Without loss of generality 
we always assumes B > along z-direction. The interac- 
tion can be decoupled into two ways by Wick's theorem: 

(H mt ) MF = U{(^H) + \{^H) 2 - ^W) 2 ), (8) 

We perform the variational mean-field study and use ho- 
mogeneous trial wave function which is the ground state 
of: 

9 2 

HMF=H + m^o- z ip + — + U(p+!j), (9) 

where m and p are the variational parameters; p = 
(ip'ip — 1) and m = —U(ip'a z tp) are the self-consistent 
conditions to minimize energy. Note that p is the elec- 
tron density with respect to half-filling. 

Hq +mil)^cr z i/j gives the well-known Landau levels (LL) 
of a massive Dirac cone. The energies of LLs are: 



sgn(n) x j2\n\-j^- + m 2 , if n ^ 

if n = 0. 



(10) 



Define (Hq 

(A 2 +m 2 ) 3 / 2 

6-7T 



m^o- z ip)\ B = 
A(m) 4 



/■A \j p 2 +m 2 dp 
JO 2tt 
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where a hard mo- 



mentum cut-off A is introduced. Physically, A is order of 
surface band width. A(m) is a cut-off dependent regular 

function which is A(m) = — ^ A + ^ - 1 — with the above 
simple rcgularization of a hard cut-off at A. In the fol- 
lowing we show that general analytical expression can be 
obtained: 



About Eq. flU) 

Eq.Q plays a key role in our study and we will prove 
its validity in this section. First, as far as we are aware of, 
the correct form of the energy function is written down 
as in Eq. (j4} for the first time. The major difference 
between the current result and the previous results is 
that, when p — 0, the energy has no linear dependence 
of m. The previous results [161 ] are incorrect because the 
rcgularization scheme is wrong. 

Now we derive Eq. (j4]). In an external magnetic field, 
we consider the orbital effect and ignore the Zeeman cou- 
pling for the moment since adding its contribution to en- 
ergy function form is straightforward. The Hamiltonian 



(H + mip'a z tp) - A(m) 
--(H + mip^a z ip) - (H + m7/i t o- z -0)| B=o +- 



Gtt 



mp - 



B\m\ 
2tt 



(2Bf 2 ,1m 2 , 

- — - — C — , — , if 



which is another way to write Eq. 

The coefficient of the second term of Eq. (jlip is twice 



the previous incorrect result in Ref. Il6l The previous 
calculation basically sums over all filled LLs whose en- 
ergies are given by Eq. (fTO]). and then substract out 
an infinity to obtain the finite result. If this would be 
the correct regularization scheme one would expect that 
when v = —1/2 the energy is independent of the sign 
of m because all the negative LLs' energy only depends 
on m 2 . Interestingly, we will prove rather convincingly 
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below that the energy function obtained in this way in- 
correct. 

For any finite size lattice, the magnetic field only cou- 
ples to hopping t — > te w , indicating that the trace of the 
hamiltonian is invariant with or without magnetic field. 
This means that in the presence of particle-hole symme- 
try (which is true for both our lattice calculation below 
and the low energy effective theory in magnetic field), 
the summation of all energy levels remains zero indepen- 
dent of magnetic field. When magnetic field is turned on, 
the energy of the Oth LL is —m. This means that, the 
LLs with index ±n cannot have the same energy mag- 
nitude as shown in Eq. (fTU)) , otherwise the trace of the 
Hamiltonian would be —to. Let us use E—(m) to denote 
the summation of the energies of all LLs with negative 
indices (n = —1, —2, • • • ) and use E + (m) to denote the 
summation of the energies of all LLs with positive indices 
(n = 1, 2, • • ■ ). The conservation of trace of Hamiltonian 
dictates: E+(m) + £'_(to) — to = for arbitrary B. This 
means that at least one of E + (to) and E- (to) has linear 
term depending on to. In fact we find they each has a lin- 
ear term to/2. Only when this linear shift is considered 
the regularization scheme can be correct. 

As we mentioned, the correct regularization scheme is 
essential to obtain Eq. (fTTjl . One physically correct regu- 
larization is to treat the Dirac fermion as the continuum 
limit of a lattice model. We perform a numerical calcu- 
lation of a lattice Dirac fermion model which confirms 
the validity of Eq. (fTT|) . Even thought it is true that 
it is impossible to realize a single gapless Dirac fermion 
in 2D by the no-go theorem, we can consider a quantum 
anomalous Hall mass in a lattice model with two gapless 
Dirac fermions, whose energy function is just twice of the 
single Dirac fermion case. Since the p dependence in Eq. 

is all from -mp, in the following we only need prove 
that Eq. (JTTJ) is correct for the case of p = i.e., 
v = —1/2 or a fully empty zcroth LL. 

We consider a time-reversal symmetric two- 
dimensional square lattice tight-binding model with 
two gapless Dirac cones [joj]. and then put in the 
quantum anomalous Hall mass term as follows: 

Hutt. = ^2 ^\2t{T x a v s\nk x + T x cr x smk y )$> k 



+ ^{(a + &(cos k x + cosfc y ))r 2 * fc + m*£(-o-*)* 



fc- 



(12) 



In this model there are four spin-orbital coupled states 
per site: {s+, s_,p+,p_}, and a and r matrices are act- 
ing within the spin and orbital spaces respectively. Only 
on-site terms and nearest neighbor hoppings are consid- 
ered. Let m = for the moment. We set t = 0.5 and 
a = —2b = 0.5 so that at half-filling there are two degen- 
erate two-component Dirac cones at L point with vp = 1. 
Now we turn on the quantum hall mass to. Depending on 
the sign of m the 2D system is in either v = \ ox v = —\ 




FIG. 3: Lattice calculation of various system sizes compared 
with gf!L-i/2( x ) based on Eq. The inset studies the 

finite size scaling at x = 0, which is consistent with the leading 
correction to scaling. 



integer quantum hall states. We then turn on a orbital 
magnetic field B by modifying t — > te la and calculate the 
energy keeping the two Oth and higher LLs empty but 
lower LLs fully filled. By defining and computing 

1 Iml 3 
£U_ 1/2 (m,5) = ^[(H iatt .)\B - (H latt .)\ B=0 + L-L] 



(13) 



where the factor of k is introduced because we should 
compute the contribution from only one node, we will 
show that it agrees with Eq. (|11|) . 

We use periodic boundary condition and insert L flux 
quanta uniformly over a sample of L x L lattice such 
that the magnetic field is B = 2-k/L. We expect 
that, at low energy, E„ = _ 1 / 2 (m, B) = g v= -i/ 2 {x)B 3 / 2 , 
where x = m/yJ~B, by the scaling analysis. We nu- 
merically compute g v =-i/2{%) for various system sizes 
L = 100, 200, 400, 800 as shown in Fig. [3] If Eq. (JTTJ is 
a correct form for energy function of a single Dirac node, 
we should be able to see that, as L — > oo, g v =-i/2( x ) 

\x\ _ 2 3/2 /-I 1 
2tt 4i ^' 2 



)• 



should approach ff^L_ 1 / 2 ( x ) = 
the result based on Eq. (fTTj) . 

When L goes to infinity, the numerical result from the 
lattice calculation indeed converges to the analytical re- 
sult of Eq. (jll|) , as shown in the Figj3l The inset studies 
the finite size scaling where the deviation from thermo- 
dynamic result is nicely fit by oc 1/VZ. This finite size 
scaling is expected because when x = m = 0, the energy 
scales as ~ aB 3 ^ 2 + bB 2 + ... where the B 2 term is the 
leading correction to scaling. As the magnetic field that 
we used is B = 2t:/L, we expect the leading correction 
to scaling oc l/yL, which is consistent with the lattice 
numerical calculation. 

Finally we mention that there are several simple limits 
of Eq. (|11[) that one can understand. When to = 

^ 3 ^(47f 2 )^ — 1 wn i cn i s a well-known 



Eq. (fTTj) becomes 



result [211 l22|. When B = the r.h.s. of Eq. ([TT]) 



becomes which is consistent with the second line. 
Note that the latter two terms depend on to 2 only: 



«(x)=>x+jX+j3X 

BCS ZS ZS' 




o ^ o 



- 0.02 | J - 0.4 

. i v I 

2 2 

FIG. 4: We plot the optimal mass m,(v) in units of _B 1//2 and 
the leading term of mean-field energy H^ngiy) = f(f)B 3 ^ 2 in 
the quantum critical regime. The non-convexity signals phase 
separation. 

" ^^a-h §£)] oc m 2 for m/VB « 1 so that 
the first term of is the only term which couples p and m 
linearly. The scaling behavior in Eq. 73 ~ p ~ to 2 

simply follows the fact that 1/m and are the only 
length scales when [/ is absent. 



The large-TV analysis 

Now we include quantum fluctuations by a large- TV ex- 
pansion. The mean-field study that we performed can be 
reinterpreted as the leading order of large- TV expansion 
of the following generalized model: 

H = ipl[IL-a}tp a -G m + p 2TV 1 ^ ' 

where repeated indices means summation over TV flavors. 
We tune G m to reach the quantum criticality G m ^ c in 
the large- TV limit, beyong which a mass m£ , il>\o ~ z tj) [ a 
term is generates dynamically and which is controlled 
by a Gross-Neveu universality. G p is irrelevant at this 
critical point and the free fermion fixed point and can be 
treated perturbatively. Quantum fluctuation is included 
if we consider a finite TV. However whether the critical 
point G myC is stable at the physical TV = 1 is unclear. It is 
worth to note that there are evidences of a stable G m c at 
TV = 1 by functional renormalization group method j23|. 
In our study we assume that G m ,c is stable so that the 
transition at U c at zero magnetic field is continuous. 

Now the relation between Hatree-Fock approximation 
and the large-TV analysis is clear. Without the 
term, Eq.Q would be identical to the effective action of 
Eq.(dU without the G p term, in the large- TV limit. The 

term in Eq.Q, similar to the G p term in Eq. (fl~4)) . is 
just a non-singular perturbation at the quantum critical 
point. 

We now show that, as shown in Fig. QJa), phase sep- 
aration occurs in regimes I and III at least when TV is 
large. On the other hand, phase separation should occur 
in regime II even when TV = 1. 



FIG. 5: The three processes of the second order perturbation. 
Black dot represents it. External legs are in the 0th LL and 
internal legs are in higher LLs. 

First in the quantum critical regime I, the singular part 
of energy still have form H S i ng = f(v)B z / 2 because en- 
ergy has no anomalous dimension, which dominates any 
regular terms including G p B 2 energy cost term. In Fig. 
Uwe plot f(v) in the large- TV limit (or at MF level). The 
non-convexity of f{v) signals phase separation. At least 
when TV is large f(y) is still non-convex and phase sep- 
aration occurs. But when TV = 1 it is unclear whether 
this function is still non-convex. In fact the more inter- 
esting scenario is that when TV = 1, the ground state in 
regime I is in a homogenous gapped fractional quantum 
hall phase. If this is the case the fractional quantum 
hall phase can be a room temperature effect, because its 
excitation gap is comparable to the LL spacing. 

Second in regime III, Ib ^ £ so the renormalization 
group flows to weak U limit where am 2 term is large 
and m fluctuation is suppressed. Thus the phase sep- 
aration energy gain cx B 2 is reliable for any TV. By 
scaling analysis H ga i n ~ £B 2 where £ diverges as crit- 
icality is approached. It wins over any regular energy 
cost terms including G p B 2 . However when TV = 1 it is 
unclear whether H ga i n ~ £,B 2 is positive or negative. 

Finally even when TV = 1 we have the following argu- 
ment for phase separation to occur in regime II. In regime 
II the phase separation energy gain scales as T3-linear (in 
particular when U is close to U Cl energy density gain 
~ ttt by hyper-scaling hypothesis.), which dominates 

any regular B 2 energy cost. Note that the 73-linear de- 
pendence is reliable when quantum fluctuation is present, 
because at the length scale of Ib the RG flows to the 
deeply ordered Gaussian FP where the mass magnitude 
|m| is not fluctuating. 

Second order perturbation 

In the presence of magnetic field the single Dirac node 
form landau levels whose wave functions can be solved 
neatly. 

If we define = =^ (TL X + iU y ), then [a, a 1 "] = 1 and 
in this representation H n = ^ [ v j' h ■ Apart 

from a, there is another set of variables describing the 
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different states within the same landau level. Following 

I — 1 i 2 

MacDonald[24|, one defines b = (z + i-f-^) wnere 

z = x + iy and II = Tl x + iH y . It is easy to show that 
[b, ¥] = 1 and [a, b] = [aJ, b] = 0. These define a basis of 

wavefunctions: \n,m) = ^-^==— 10, 0) where a|0, 0) = 
6|0, 0) = 0. And the energy levels and eigenfunctions are: 



I*, 



V2' 



V2 

E n = sgn(n)y / 2|n 



sgn(n)i|n — 1, m) 
\n, m) 

Hvf 



(16) 



where sgn(n) is the sign function and sgn(0) = 0. 
The full hamiltonian is H = Hq + Hi nt where: 



Hint — 



dq 2 

(2^) 2 

l<3 V ' 



V(q), 



iq-(ri-rj) 



(17) 



For pure Coulomb interaction V(q) = ^o - - In the second 
quantization formulism: 

H E n ip^ nrn ip nrn 

+ \ V 4-.3^L,ra i ^i 3 ,rn 3 i J n 2 ,rn 2 'lpn 1 , mi (18) 

ni,rrii 

where V4 3-2 1 is an abbreviation of: 



v, 



TI4 ,m4 ,773 ,7773 ; 77 2 ,777,2 ,77i ,777,1 
2 



dq 2 



(2*) 



> r \^ /\ 714 ,777,4 I \ 713,7713 I ^ I ni,J7ll/l TJ2 ,7712 / 



-|QI : 



72' 



" C77I4, 777-1 (-Q)G 7713 ,7772 (Q) 



•[sgn(n 4 )sgn(m)G ! |„ 4 |_ lj |„ 1 |_ 1 (-(9) + G|„ lMlll |(-Q)] 
■[sgn(n 3 )sgn(n 2 )G ! | T[ 3|_i ) | n2 |_ 1 ((3) + G| 

"3 I, I "2 I (Q)]- 

(19) 

Here Q = (fe + «%) • and G„ lt n 2 (Q) is related to 
generalized Laguerre polynomial L™;j,~" 2 : 



Cm ,ti 2 (Q) 



"1! 



V2j 



V 2 



(20) 



Note that for Coulomb interaction V^^.i <x jj-^. 

Now we antisymmetrize by defining Vi.s,-2,i = 

V4,3 ; 2,i — V3A;2, 1) an d S° to action formulism, Z = 
/ P^DI-e 5 :' 

77,777, 

/il^ 1 E U 4,3;2,l^ 4 ,m 4 (w4)^3, m3 (a;3) 



1 /" rr <iw,: 
4.. 

(w 2 )V'n 1 ,m 1 (wi), 



(21) 



where W 4! 3 ;2 ,l = t4,3;2,1^4,3;2,l an d ^4,3;2,1 = 2 7r5(w 4 + 
U) 3 - w 2 - tJx). 

Next we integrate out fermion modes with n =/= 
in a single step and obtain a renormalized interaction 
1*4,3-2 1 + rfu4,3 ; 2,i in the 0th LL. 



du 



4,3;2,1 



"i=0 



. , m 5 .m e 

dui^duiQ 
(2tt) 2 2- 



1 



-"6,5;2,lW4,3;6,5 



+W6,4;5,lW3,5;2,6 — W6,3;5,lM4,5;2,6] ^5^6, 



(22) 



The three terms can be sum- 
1 _ 



where ft = 

marized in FigfS] Using identity J 

e (Re(A)) : HRe(B)) whcrc ^ = j . f ^ > Q ^ ^ 

otherwise, we perform the frequency integration: 



(iw- J 4)(iw-B) 



(/(/ 



4,3;2,l|„ i=0 
m 5 ,m 6 



715,77,65^0 



i f/ f/ e(^ ne )-g(-£; ns ) 

r K6,5;2,l l/4,3;6,5 T7 ; r - — 



+ V6,4;5,lV3,5;2,67 
— Vj3,3;5,1^4,5;2,6 



0(E ns ) - 9(E n6 ) 



i(u 2 - cj 3 

0(E ns ) - 0(E ne ) 
i(Lo 2 - UJ4) + E ne - E n5 



^3;2,l (23) 



The retardation effect gives the frequency dependence 
which can be neglected for low energy dynamics when 
relevant energy scales are much smaller than the energy 
gap, namely u <C E\ . This allows us to go back to Hamil- 
tonian formulism. 

We finally present the renormalized hamiltonian in 
terms of the well-known Haldane's pscudopotcntials. 
Because the interaction pcrserves the relative angu- 
lar momentum of two particles in the same LL, one 
can parametrize the effective interaction as: V m = 
(m, McM\Hi n t\m, Mcm) where m, Mqm are the relative 
and total angular momentum of the two particles respec- 
tively. The V and peudopotentials V m are related by: 

y\ Ki4- 1,7713— 0;m2— 0,mi — 1 

V3 = J (V3,0;0,3 + 3V2,1;1,2 - V3(F 3 ,0;1,2 + V 2 ,l;0,s)) (24) 

where in the second line we abbreviate the m, labels. We 

2 

find for the 0th LL in units of 4—: 

el B 



Vi 

v 3 



yCouU 



dVi = 0.443 - 0.0068a 

dV 3 = 0.277 + 0.0028a (25) 



If we trust this leading order perturbation result even for 
large a, and using the dV\ ~ dV^ phase diagram obtained 
by exact diagonalization[18j, we find that when ^5 a < 
4 the system is in CFL phase, when 4 < a < 7 the 
system is in Pfaffian phase and when 7 < a the system 
is in Stripe phase. 
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The Zeeman effect 

The Zeeman coupling between electron spins and mag- 
netic field is given by 



H 2 



1 



-g\i B Bo z 



1>, 



(26) 



where g is the (/-factor. The Zeeman coupling simply 
renormalizes the mass term: m —> m = m+^gpsB in the 
energy function Eq. By denoting p = p + ^gpsB/U, 
the MF variational energy is given by 



E(p, m; B) = —rhp 



B\m\ 



(2B) 



3/2 



2tt 

(m 2 + A 2 ) 3 / 2 

()7T 



4tt 

1 



a- 1 - —) 



- ~2 , U P 2 / 07 N 

— m H , (27 

2U 2 ' V ' 



which has the exactly same form as the equation without 
Zeeman coupling. Note that the difference is that (p) 
averaged spatially is p„ = gpsB /2U instead of for zero 
magnetic field. 

Suppose that Eq. (|2T[) is minimized by an optimal 
mass is m* > if the system is homogeneous namely 
p = p* everywhere. To see if phase separation is favored, 
we consider the situation that the system macroscopically 
separates into two regions with the configuration of (p = 
p* + 5p, m = to* + 5m) and (p — p„ — Sp, m = m* — 5m) 
respectively with small Sp and 8m. Then its average 
energy density is given by 

(E) = [E{p*+8p,m* + 5m)+E{p*-8p,m*-8m)}/2 
= E{p„,m„;B) + -F"(rh*)Sm 2 



+ ^{8p - Sm/Uf + 0{Sm 4 ), 

where F(rh) is defined as 

F (~) - W\ { 1 ™L) _ (m 2 +A 2 ) 3/2 
4tt ^ 2' 2B' 6tt 



(28) 



(29) 



From Eq. (|28|) . it is obvious that Sp = Sm/U is required 
to minimize the energy function (E). Consequently, as 
long as F"(m*) < 0, phase separation will be favored. 
Explicitly, F"(m^) is given by 



F"(m*) 



{2Bf/ 2 



An 



^ { 2 , 2B ) 2B^ { 2'2B' 



~ [(^ + A 2 ) 1 / 2 + - 1 
27T [m* 



~ 2 

mi 



A 2 )V2 



-in 



2{x + y) 1 / 2 



' (x + y) 1 / 2 



(30) 



where x = m 2 /2B and y = A 2 /2B ~ l 2 B /ar >• 1 for cur- 
rently available magnetic field. We know that m* ~ B 



since a non-zero m„ is induced by magnetic field. Thus, 
x = rh 2 /2B ~ B; for weak magnetic field, x <C 1. Be- 
cause x((^,x) — ((^x) is a monotonically increasing 
function and its values ~ [1-5,4] for x £ [0,1], we ex- 
pect F"(m») oc [0{1) - 21 B / a] where O(l) is order of 
one. Consequently, the condition for F"(m t ) to be nega- 
tive namely tendency to phase separation is equivalent to 
Ib 3> a, which is always satisfied for currently available 
magnetic field. In other words, the Zeeman effect has 
no qualitative effect on the tendency of phase separation 
when only short-range interactions are considered. 

What happen when long-range Coulomb interaction is 
included? What are the fates of the stripe phases in 
regime I and II predicted without Zeeman effect? Be- 
cause Zeeman coupling does not give new length scale 
and can be viewed as an redefinition of v. v — > v = 
v + 7rg ^: B . This means that the energy gain in the regime 

I still scales as ~ B z l 2 (it is a gain at least in the large- TV 
limit). So the predicted stripe phase in regime I is as 
robust as the case without Zeeman coupling. 

On the other hand, the "axionic stripe" phase in regime 

II may be destroyed by Zeeman effect. "Axionic stripe" 
phase requires that the phase separation energy density 
gain is linear in B, which in turn requires that the masses 
in the v = 1/2 and v = —1/2 regions have opposite signs. 
As mentioned in the main text, the condition for this to 
happen is 47r ^ B < 1 (U > U c and U c - 2nhv F /A), oth- 
erwise both regions have the same mass sign so that the 
phase separation energy density gain scales as B 2 and 
phase separation will be destroyed by Coulomb interac- 
tion. 

Let us estimate the ratio ^Ms_. 



eU c 



AirhgpB 47T hg^-A _ hA 



eU c 



e2nhvF 



9- 



m e VF 



A e 

9— > 
a 



(31) 



where A e is the de Broglie wavelength of the electron 
moving at vf and a ~ 2n / A is the short distance cut-off 
which is comparable to the lattice spacing. This is only 
an order of magnitude estimate because, for instance, we 
use the mean- field value of U c and the cut-off A cannot be 
accurately estimated. If we nevertheless plug in vf ~ 3 x 
10 5 m/s (as in Bi 2 Te 3 ) and a = 8 A, we find - 3 5 , 

which is of order one. We thus expect in systems with 
large Dp, a and small g, this ratio is smaller than one and 
the "axionic stripe" phase can be realized. 

Finally we comment on the magnitude of (/-factor for 
the surface states of TI. It is known that in TI systems 
with strong spin-orbital interaction, the (/-factor in the 
bulk can be as large as 30~50 (for example, Bi2Se3 has 
g ~ 30 in the bulk[5[-). If the (/-factor on the surface is 
as large as in the bulk, the Zeeman coupling can be a 
large energy scale. However, it is not obvious that the 
(/-factor on the surface is comparable with the one in the 
bulk. In fact, it is observed by STM experiments 0, H[ 
that the 0th LL energy of the surface state is basically 
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field independent up to ~ llTcsla (the change of 0th LL 
energy is within a couple of meV as held is tuned from 
3Tesla to UTesla). We conclude that the g-factor on the 
surface cannot be ~ 30, which would mean that the 0th 
LL energy shifts ~ 15meV if field is tuned from 3Tesla 
to UTesla. This is a direct evidence that the surface g- 
factor can be an order of magnitude smaller that in the 
bulk. 
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